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ABSTRACT 



_The original purpose of this L study was to address the 
test-disclosure-reiated need to introduce more Graduate Record 
Examinations (GRE) General Test editions each year than formerly, in a 

context of stable, or possibly declining examinee volume. The 

legislative conditions that created this initial concern regarding test 
equating have abated. However, several of the test equating models 
considered in this research might provide other advantages to the GRE 
Program. These potential advantages are listed in the body of the 
report* 

Equating can be considered to consist of three parts: (1) a data 

collection design, (2) an operational definition of the equating 

transformation, and (3) the specific statistical estimation techniques 
used to_ obtain the equating transf ormatibrii. _ Currently, the GRE General 
Test collects data using an equivalent groups design. Typically, a 
linear equating method is used, and the specific estimation technique is 
setting means and standard deviations equal. 

For this research, two other data collection designs were studied: 
nonrandom group , external anchor test^ and random_group* preoperational 
section. Both item response theory (IRTJand linear equating definitions 
were used. IRT true score. equating was based on item statistics for the 
threerparameter logistic model as estimated using LOGIST. Linear models 
included section pre-equating using the EM algorithm, Tucker's observed 
score model, and several true score models developed by Tucker and 
Levine. For each of the three GRE measures, verbal, quantitative and 
analytical, all equating methods were assessed for bias and root mean 
squared error by equating a test edition to itself through a chain with 
six equating links. 

---- -Bias and root mean squared error were extremely large for equating 
the yerbal and analytical measures using section pre-equating or IRT 
equating with data based on the random group preoperational section data 
collection design. For the quantitative measure, this data collection 
design /produced a small amount of bias, but moderate amount of root mean 
squared error. 

____ _Using„the nonrsndom group, external anchor test data collection 
design, quantitative equatings had moderate amounts of both bias and root 
mean squared error. Verbal nonrandom group , external anchor test 
equatings showed relatively small amounts of bias and root mean squared 
error, with the Tucker observed score model performing particularly well. 
Bias was small for the analytical anchor test equatings, and root mean 
squared error ranged from small to moderate i 

All_ nonrandom group, external anchor test methods worked about as 
well in practice for the verbal measure as the currently used random 
group method does in theory. The current random group method, however, 
has never been subjected to an empirical check comparable to that used in 
this study for tbe experimental equating methods. Two anchor test 



methods, Tucker 2 True and Levine* appear to have worked as well in 
practice for the analytical measure as the random group method does in 
theory. 

A possible explanation for the generally poor results for the 
random group, preoperational section data collection design based 
equatings was the constant use of tne last section of the test to collect 
equating data. It may be, now that the sections of _ the GRE General Test 
are administered in various orders in different editions of the test, 
that the extreme bias found in this study for the verbal and analytical 
random group preoperational section equatings will disappear or at least 
be substantially diminished. 
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INTRODUCTION 



feirpose- of this study 

The original purpose of this study was to address the test- 

disclosure-related heed to introduce more Graduate Record Examinations 
(GRE) General Test editions each year than formerly, in a context of 
stable, or possibly declining examinee volume. Since then, the legisla- 
tive conditions that created the initial concern regarding test equating 
have abated. However, several of the test equating models considered in 
this research might provide other advantages to the GRE Program, such as: 
improved test security, greater accuracy of equating, shorter timer 
schedule requirements for score reporting, additional test analysis 
information, and possible improvement of the test development process. 

Equating. 

Test developers usually try to make the various editions of a test 
interchangeable with regard to content coverage, item format, and 
difficulty so that examinees are neither advantaged nor disadvantaged by 
the edition of the test they happen to take. Unfortunately, because of 
the large number of constraints imder which test developers operate and 
because of the quality of the statistical information that is available 
at the timeeditionsof a testare constructed, inevitably, some test 
editions are easier than others. To make sure that groups of examinees 
taking different editions of a test are treated equitably, statistical 
techniques known as test equating are used to adjust scores on each 
edition of the test so that they are comparable to scores earned on other 
editions of the test. 

There are several_different equating models used by 
psychbmetricians. _ Thesemodeis make different assumptions about the data 
they use and vary in their appropriateness for any given examination, 
such as the verbal, quantitative, and analytical measures of the GRE 
General Test. There are three major aspects of any equating model: (1) 
the data collection design, (2) the operational definition of the 
equating transformation, and (3) the specific statistical estimation 
techniques used to obtain the equating transformation. 

Data collection designs. Two data collection designs were used 

specifically for this study. For the first design, referred to as 
NonRandom group External Anchor Test (NRRAT) , two editions of a test were 
administered , one to each of two nonrandom groups consisting of examinees 
who chose to take the test on one or the other of two test administration 
dates , and a common short test that did not count toward the examinees* 
scores was administered to both groups. This design is sometimes 
referred to as Design IV from Angoff (1984). 



z - For the second design, referred to as Random Group, Pre-Operational 
Section (RPOS) , essentially , one test was administered to a group of 
examinees • That group was further divided into two equivalent subgroups 
through the spiraling of test booklets. That is, different versions of 
that test edition were packaged in an alternating fashion (e.g., 1,2; 
1|2; 1*2; ...). Research has shown that spiraling results in 
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essentially equivalent groups * sometimes even more effectively (because 
of a stratification effect). than does true random assignment. In 
addition 9 one subgroup received one-half of a second edition of the test, 
and the other subgroup received the other half of the second edition.. 
The two half-tests were designed to be as similar as possible in content 
and difficulty. For further information on this design, see Holland and 
Thayer (1981, 1985), Holland and Wightman (1982), or Petersen, Hoover, 
and Kolen (in press). 

The data collection design used currently for operational GRE 
General Test. equatings is referred to as Random Groups (RG). Two 
editions of the test are given, one to each of two random, or otherwise 
equivalent, groups. The GRE Program regularly uses spiraling to ensure 
equivalence of the two (or more) groups. This design was used in this 
research to provide some comparison equatings. 

More detail regarding the data collection designs used in this 
study is given in the Procedures section of this report. 

Equating - trans format ions . Three operational definitions of the 

equating transformation are commonly used: (1) linear equating, which 
provides a transformation such that scores from two tests vtli be 
considered equated if they correspond to the same number of standard 
deviations from the mean in some population of examinees, (2) 
equipercentile equating, which provides a transformation such that scores 
fromtwo tests will be considered equated if they correspond to the same 
percentile rank in a specified population of examinees, and (3) item 
response theory (IRT) equating, which provides a transformation such that 
scores from two tests will be considered equated if they correspond to 
the same level of the latent trait underlying the two tests. Only linear 
and IRT transformations were used in this study. 

------ Statistical estimati on techn iques . A number of different 

techniques have been developed to estimate the intercept and slope, 
parameters for a linear equating. Each technique attempts to estimate 
the first two moments of the score distributions for an old edition (one 
whose scores are already on scale) and a new edition of the test on some 
common group of examinees. These estimation techniques differ in the 
assumptions that they_require. Aprimary difference is that some linear 
methods estimate. the means and standard deviations of observed scores arid 
bthersestimate the true score moments- Estimating true score moments is 
considered particularly appropriate when the two editions of the test to 
be equated have been administered to groups with very different ability 
distributions (Angoff , 1984, p. 113). In this study, various statistical 
estimation techniques, which will be described later in this. report, were 
used with the NREAT data collection design^ and one, algorithm, to 
estimate the rirst two observed score moments (Holland and Wightman , 
1982) was used with the RPOS data collection design. This latter method 
is commonly referred to as section pre-equating, or SPE. 
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_ This study investigated only one IRT equating method, IRT true 
score equating (Lord, 1980, pp. 199-200). There are three aspects of 
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statisticalestimatibn for IRf true score equating: (1) estimating item 
and person parameters, (2) putting the parameter estimates from separate 
calibration runs on a single scale, and (3) setting, equal the true scores 
that correspond to the same level of the latent ability, theta. 

The GRE General Test 1 

The GRE General Test measures andyields separate scores for the 

general verbal* quantitative, and analytical abilities students should 
have acquired to be successful at the graduate level of education. 
Scores for each measure are based on the number of correct answers and 
are scaled to fall between 200 arid 800. The test consists of seven 
30-miivite sections of multiple-choice questions* At the time datawere 
collected for this research, sections i and 2 constituted the verbal 
measure; sections 3 and. 4, the (quantitative measure; and sections 5 and 
§1 analytical measure. The remaining section does not count toward 
any of the reported scores, and usually consists of verbal, quantitative, 
or analytical pretest items. For this research, the remairiirig section 
was used to collect data for the equating experiments. The specially 
constructed versions of this section are described in the Research Design 
section of this report. Since the data for this research were collected, 
the ordering of sections of the General Test has changed. For current 
editions of the GRE General Test, the seven sections may be arranged in 
various orders. 

The verbal measure employs four types of questions: antonyms, 
analogies, sentence completions , and. reading comprehension sets. The 
quantitative measure employs three type of questions: discrete 
quantitative questions, data interpretatibnquestibnSi and quantitative 
comparison questions • The_ quantitative questions measure basic 
mathematical skills* understanding of elementary mathematical concepts, 
and. the ability to reason quantitatively and solve problems in a 
quantitative setting. These questions require arithmetic^ algebra* arid 
geometry at a level not beyond that taught in a first high school level 
course. 

The analytical measure emplbys_twb_types_of questions! analytical 
reasoning. and_ logical reasbningi Anaiyticai reasoning questions test the 
ability. to understand a given structure of arbitrary relationships among 
fictitious persons, places, things, or events; to deduce new information 
from the relationships given; and to assess the conditions used tb : 
establish the structure of relatibriships. Logical veasbriirig questions 
test the ability to understand, analyze, arid evaluate arguments: _ 
recogriizirig the assumptions on which an argument. is based* .drawing 
conclusions from given premises* inferring material missing from given 
pass* ges^ .applying to one argument principles governing another, 
identifying methods of argument, evaluating arguments and 
counter-arguments, and analyzing evidence. 



*This section of the paper was adapted from the Guide to the Use of the 
Graduate Record Examinations Program 1985-86 (ETS, 1985a). 
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Additional information on the content of the GRE General. Test and: 
examples of the various item types c«n be found in the GRE B ulletin (ETS, 
1985b). 
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RESEARCH DESIGN 



In this section the database (test editions and examinee samples) 
is described , arid the various procedures used in this research are 
detailed. The equating modelsused and the assumptions upon which they 
are based are explained. The rationale for the criterion used to judge 
the adequacy of the equating models is developed. 

Database 

Six editions of the GRE General Test were administered on seven 
different occasions; the edition given at the first and last 
administration was the same. 



Test editions . For ease of reading, the six editions of the GRE 
administered as part of this research will be referred to in this report 
as El, E2, E3, E4, E5, and E6. (The ETS designations for these test 
editions are 3DGR3 , 3DGR1 , 3DGR2 , 3EGR1, 3EGR4, and 3EGR2, respectively, 
for the verbal and quantitative measures. For the analytical measure, 
edition 3EGR3 was used instead of 3EGR4.) One of several different 
experimental sections was administered along with each edition. These 
experimental sections were used as either anchor tests for the NREAT 
equating data collection design or as pre-operational sections for the 
RPOS data collection design. The use of these data for equating will be 
further explained later in this section of the report. 

Table 1 describes the characteristics of the verbal, quantitative, 
and analytical measures for each test edition. It shows when each 
edition was administered as part of this research. For each measure it 
gives the. number of items. contributing to the reported score for that 
edition and the mean and standard deviation of the difficulty of the 
items in that measure. In addition, the number of items in each 
experimental section and their difficulty are presented. Appendix A 
presents (among other information) the ETS form and subform designations 
and codes for each test. 



Insert Table 1 About Here 



Examinee samples . Samples consisted of all examinees who took the 
appropriate test editions at one of the seven administrations, with the 
exception of all of the following: 



examinees who had taken the GRE General. Test more than once and 
who received any of the same test sections at two or more 
administrations , 



Item difficulty is presented in terms of equated deltas, that is deltas 
put onto a common scale for a given measure for all test editions. See 
Henrysson (1971, pp. 139-140) for a description of the delta statistic. 
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ci examinees who did not respond to at least five items in each 
of the seven sections of the test, 

• examinees without an item response record (i.e.* any examinees 
whose answer sheets were not machine scored), and 

• examinees who took the test at a center for which an 
administrative irregularity was reported^ 

Tables 2, 3, and 4 present information regarding the examinees 
tested as part of this research — the sample sizes and means and 
standard deviations of their scaled verbal, quantitative, or analytical 
scores for the subgroups tested at each. administration* The statistics 
are based on only those examinees used in each equating. Appendix A 
presents (among other things) the number of examinees in the sample for 
each external anchor test and each preoperational section. Anchor test 
samples ranged from 3,583 to 4,408 examinees. Preoperational section 
samples ranged from 1,745 to 2,561. The approximately two-to-one ratio 
of sample sizes for anchor test and preoperational section samples was 
planned, since two preoperational section samples, but only one anchor 
test sample f are needed for each score being equated at a given test 
administration. Note, however, that all data to be used for equating 
based on preoperational sections can be collected in one administration, 
but data for equating using external anchor tests must be gathered from 
two test administrations. 



Insert Tables 2, 3, and 4 About Here 



Figures 1 and 2 present the NREAT and RPOS data collection designs , 
respectively. For the NREAT design, at each administration (other than 
the first and last) six forms of the edition administered were spiraled, 
two for each of the General Test measures. In each pair, the operational 
test edition was the same, but one form contained in the seventh section 
an anchor test (containing items of the same. typesas. the measure being 
equated) in common with the previously administered test edition, and the 
other contained an anchor test in common with a test edition scheduled 
for a future date. 

= = z:i: For the RPOS data collection design, at each administration six 
forms of the test were spiraled, two for each of the three General Test 
measures . In each pair the operational test edition was the same, but 
one form contained in the seventh section one of the two operational 
sections of either the verbal, quantitative, or analytical measure from a 
previously administered edition, and the other form contained the other 
operational section. Note, this is the opposite of what is normally done 
for an RPOS data collection design. Usually, these sections would 
contain halves of future editions instead of previous editions. 
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Insert Figures 1 arid 2 About Here 



Procedures 

■ Equating methods. Results from seven different equating methods 
are presented in this report — one RG method: setting means and 
standard deviations equal; four NREAT methods: Tucker, Tucker True 2, 
Levine (equally reliable, or unequally reliable, as appropriate), and IRT 
Anchor Test True Score with theta metric set using concurrent 
calibration; and two RPQS methods: IRT Preoperational True Score with 
theta metric_set using concurrent calibration, and EM algorithm to 
estimate means and- standard deviations. Three-parameter logistic IRT 
estimates were performed with the program LOGIST (Wingersky, Barton, & 
Lord, 1982), A brief overview of these methods is given in Table 5, and 
a detailed description of the linear methods is presented in Appendix B 
(Appendix B is adpated from appendix A of Marco^ Petersen* & Stewart, 
1983). Detailed information on IRT true score equating is available in 
Lord (1980, chapter 13)_and Hambleton and Swaminathan (1985, chapter 10). 
Information on the use of the EM algorithm for equating with RPOS data 
collection designs is available in Holland and Wightman (1982). 



Insert Table 5 About Here 



Some additional_linear equating methods were used in early stages 
of this study as was a second method of establishing a common IRT metric. 
Appendix 6 presents some notes on these methods. 

Assessing the adequacy of equating methods. A good equating method 
should have certain characteristics. As with many other statistical 
estimation techniques, these desirable characteristics include minimal 
bias and mean squared error. Assessing the bias and mean squared error 
of one ormore eqaatingmethods, however, first requires one to know the 
true equating relationship between test editions. In most real-life 
equating situations, this is riot possible. For the purpose of this 
research, such a criterion was constructed. 

For each equating method in this study, a chain of six equatings 
was performed. E2 was equated to El, E3 was equated to E2, E4 to E3, E5 
to E4, E6_ to. E5 1 and_ then finally El was administered again arid equated 
to E6. If the function that equates E2 to El is called f(x), and the 
function that equates E3 to E2 is g(x), then the composite function 
g(f(x)) will put scores from E3 on the El scale. Likewise, if h(x) 
equates E4 to E3, i(x) E5 to E4, etc., then k( j(i(h(g(f (x)))))) equates 
El to El through the chain (or circle) of six equatings. Since the 
equating of El to El should be an identity function, it can be determined 
how close each equating method came to the true equating relationship. 
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The equating criterion was used _ in several ways. First; equatings 

were compared graphically* These graphs were summarized statistically. 
In doing so, it was decided that inaccuracies in equating were 
inconsequential if few or no examinees were affected by them, arid so 
discrepancies were weighted at each raw score by the number of examinees 
(whose data for edition El was used in this research) who obtained that 
score. Two summary statistics were calculated; bias (equation 1), the 
weighted mean_difference_ between an experimental equating and the 
criterion equating* and root mean squared error (see equation 2) , 
equivalent to the weighted mean standard error of equating. Note that 
root mean squared error includes bias and thus can never be smaller than 
bias* 

bias = ([ I X f ] / E f ) - [ " X.if.l / " f (1) 

i=i " 1 i=i 1 i=i 11 1 i=i x 



- (x 2i ) - x 1± 



- d 



=V ( I 



RMSE =V ( E [X - X ■ ) f + ) / E f; (2) 
i=l 11 1 i=l 1 



/n h 
( E d,fj / E 



i-i' 1 " 1 ' '±ii fi 



where n is the maximum obtained score for the measure > 

X 2i is the score (equated through the six-link chain) 

corresponding to raw score i for the April 1983 administration 
of form El, 



f ^ is the frequency of raw scores i in the October 1981 group, 

Xjj is score corresponding to raw score i for the October 1981 
administration of form El . 



In addition to the calculation of hias and root mean squared error 

on the raw score metric, to facilitate comparisons of equating methods 
across the three General Test measures , bias and root mean squared error 
(RMSE) were standardized by dividing each by the scaled score standard 
deviation for the appropriate test score. Also, to provide a context 
familiar to most score users, bias and root mean squared error were 
transformed to the appropriate GRE scaled score metric (verbal* 
quantitative, or analytical) by multiplying them by the slope of the 
scaling function used to place raw scores for edition El on the GRE score 




scale. For the verbal measure this, transformation is nonlinear, so. a 
linear approximation was used. This approximation differed from the 
actual scaling primarily at the high end of the scale where there are few 
data. 
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RESULTS 



-- --- Eigures_3^ 4, and 5 graphically present the equating- lines (raw 
score new to raw score old) for the six methods used in this stiidy A for 
the verbal, quantitative, and analytical measures* respectively. Figures 
6, 7, and 8 present the raw score differences between each equating line 
and the true equating function. So* for example, fbrthe verbal measure : 
the SPE method would convert a raw score of 72 toa 66 (even thoug\ the 
equating should have. been an identity function yielding an equated score 
of_72)._ Sixty-six minus 72 is negative six, and this can be seen in 
Figure 6. Essentially, these difference graphs simply magnify the 
discrepancies between equating methods. Note that the scale for Figure 6 
is different from that for Figures 7 arid 8. 



Insert Figures 3 through 8 About Here 



Figures 3 through 8 show that for the verbal and analytical 
equatings , the two methods based on the RPOS data collection design 
worked least well. For the quantitative equatings, the graphs do not 
show a readily discernible difference in the quality of the equating 
methods. 

^ffcle Bpresents the bias and root mean squared error for the six 
equating methods for each General Test measure. Table 7 presents bias 
and root mean squared error in the GRE General Test scaled score metric, 
so that equating error can be viewed in a context familiar to GRE score 
users. Table 8 presents the standardized bias and root mean squared 
error for the six equating methods. 



Insert Tables 6, 7 and 8 About Here 



Several findings stand out in these tables . 

• Fortheverbai and analytical measures, the equating methods 
that used a RPOS data collection design (IRT and SPE) had 
relatively large bias and root mean squared error compared to 
those that used a nonrandom group external anchor test 
design. 

• For the quantitative measure, however, the absolute 
standardized bias was least for the SPE and IRT RPOS data 
collection designs; 



• The standardized root mean squared errors for the 

quantitative equatings were not too different (ranging from 
.07 to .11) regardless of the data collection design or 
equating transformation method. 
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When taking comparisons within each of the three measures 
tha'; were equated , all models using NREAT equating performed 
aboutequaiiy well for the verbal and quantitative measures. 
For the analytical measure IRT performed somewhat less well 
than Tucker True 2 and Levine (.10 for IRT* compared to .04 
for the other two methods). 



Overall* NREAT equating models did less well in terms of both 
bias. and_ root mean squared error for the quantitative 
equatings than for the verbal or analytical equatings. 

For the verbal equatings , for each equating method, the root 
mean squared error was substantially accounted for by a 

consistent negative bias: that is, items in the 

preoperational section_wefe_systematicaliy more difficult 
than when they appeared in an earlier operational section. 
This _bias was small, however, for all of the NREAT equating 
methods; 

For the quantitative equatings based on NREAT methods, the 
root mean squared error was primarily accounted for by a 
consistent positive bias. For the RPOS data collection 
design, however, the bias was small and inconsistent , and did 
not_account for substantial amounts of the root mean squared 
error. 

Overall , the analytical equatings based on NREAT methods, 
bias was small and accounted for very little of the root mean 
squared error. For the RPOS methods, a consistent large 
negative bias accounted for most of the root mean squared 
error. 



DISCUSSION 

Comparis on of empirical root mean squared error with the standard error 
of-3r chain of operational equatings 

Iri order to answer the question, "did any of the NREAT or RPOS - 
equatings work wel 1 _ enough^ u some _ context is needed. One such context is 
the_standard_errbr of a chain of equatings based on the method currently 
used to equate the GRE General Test: random groups, setting means and 
standard deviations equal (Angoff, 1984, design IA-1 , pp. 94-97). The 
standard error of equating is affected by the size of the sample on which 
the equating experiment is performed. For operational GRE General Test 
equatings - % the samples for each edition of the test range from 10,000 to 
20,000 and thus the total sample size ranges from 20,000 to 40,000. 

Verbal equatings. For the GRE verbal measure, NREAT equating 
methods worked quite well, and RPOS methods did not. The scaled score 
root mean squared error of the chain of equatings for the four NREAT 
equating methods was about 5 scaled score points. This figure can be 
compared to the standard error of a chain of six equatings for the 
operational equating method—random groups, setting means and standard 
deyiations„equal*_ The operational standard error was estimated using 
Lordls formula forthe standard error of a single RG means and standard 
deviations equating (Lord, 1950; Angoff, 1984, p. 97) and Theorem 6, by 
Braun and Holland (1982), for the standard error of a chain of equatings. 
Assuming that the slopes of the raw score to raw score equating functions 
are close to one, the standard deviation of the equated scores for each 
hew form group was 123 (the average. for. the equating groups used in this 
research) , the total number of examinees upon which each operational 
equating is based is 30,000, and test scores are normally distributed, 
then the median standard error of equating for the chain of six equatings 
is about 4 scaled score points. (The median standard error is the 
standard error at .675 standard deviations from the mean. Under the 
previously mentioned assumptions, this median is comparable to the 
empirical weighted root mean squared error.) If the total number of 
examinees_upon_whicn_each z equating was based were 20,000 or 40,000, the 
median standard error of the chain of equatings would be about 5 or 3 
scaled score points, respectively. Of course, the figures for the median 
standard error of equating depend on the previously listed assumptions as 

well as on the assumptions of RG linear equating. In practice these 

assumptions may be violated. This would probably_increase_ the empirical 
RG linear standard errors of equating to a value at _ least somewhat larger 
than the theoretically derived numbers presented here. Thus, for the 
purpose of evaluating the root mean squared error, the median standard 
errors should be considered conservatively low. 

For the two RPOS equating methods* the average verbal equating root 
mean squared error was 26 scaled score points, considerably worse than 
the estimated standard error of the chain of operational equatings. 

— - Quantitative equating^ . For the quantitative measure, all equating 
methods had about the same root mean squared error, on the average about 
13 scaled score points. Assuming a scaled score standard deviation of 
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133* a normal distributibh^ and_sample sizes of . 26;000, 30,000, and 
40^000, _themedian standard errors of equating for the chain of six 
quantitative equatings using the operational method are about 5, 4, and 4 
scaled score points , respectively, thus, the NREAT equatings dp hot 
appear to have worked as well fpr : the quantitative measure as they did 
for the verbal measure. And, although for the quantitative measure the 
RPOS equating methods worked as well as did the NREAT equatings with 
regard to root mean. squared error, and worked better with regard to bias, 
they did uotperform as well as one would expect a random groups linear 
equating to performs 



Analytical equ atings . For the analytical measure the IRT NREAT 
equating had^the largest rbot : mean squared error ^ 13_ scaled. score points. 
The linear-NREAT equating methods had root mean squared errors ranging 
from 5 to 9 scaled score_points*„ The two RPOS equatings had an average 
rootmean squarederrbr of 26 points. With a scaled score standard 
deviation of 126 and sample sizes of 20,000, 30,000, and 40,000, the 
median standard errors of the chain of analytical RG linear equatings 
would be 5, 4, and 4 scaled score points, respectively .^Therefore, the 
linear NREAT methods did reasonably well , but the IRT NREAT equatings, 
and to a much larger extent the RPOS equatings, did poorly. 

Factors That May Have Affected these Results. 

_ Only_ for theverbai measure NREAT equatings did the root mean 
squared. error appear reasonable in light of the theoretical standard 
error of the current GRE equating procedure. This might be due to any of 
at least three factors* First, the samples used in this study are 
considerably smaller than the samples used in the operational equating of 
GRE scoresbz This might be compensated for^ however^_by_the_increased 
power of NREAT and RPOS equating designs*. Second, sampling error might 
have produced. large„ root mean squared-errors in the groups used in this 
study_even though in another set of equatings the root mean squared 
errors might be smaller. Third, the root mean squared errors of the 
experimental equatings are based oil real data and not on statistical 
assumptions. The effect of real data is likely to be an increase, in the 
size of the root mean squared error of the operational procedure. beyond 
the theoretical standard error of .equating. _If the empirical root mean 
squared error of _a chairi_of random. group , means and standard deviations 
equatings.were. calculated, it might also be somewhat larger than the 
theoretical standard error of equating. This might occur because of 
violations of the assumptions of the equating model. In particular , to 
some extent examinees are advantaged if they have previously taken the 
same edition of a test (Kingston & Turner, 1984). This can occur for the 
old edition, but not for the new edition, in an RG equating* 




1 



The choice sample sizes in this study was intentional and reflects 
administratis constraints such as current administration volumes and 
pretesting needs 
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It should _be noted that if NREAT or RPOS data collection designs 
were used operationally for the GRE General Test^ it is likely that 
double-part score equating would be used. For NREAT data collection* 
this would entail Using two anchor tests in order to equate to two 
different old editions of the test and then averaging the two equatings; 
Likewise, for an RPOS_data_collection_design, a new edition of the test 
would be prebperationally equated to two different old editions, and the 
average of the two equatings would be used. Although the statistical 
properties of double-part score equating are not well understood* such an 
equating would be expected to have reduced root mean squared error and 
would be expected to reduce certain sources of bias (although not the 
sources of bias that appear to have affected the RPOS equatings in this 
study) • 

Effect of s mal l er s am ple-size^ . If the standard errors were 
calculated for the equatings performed in this study, then the effect of 
the smaller samples used in this research could be addressed directly. 
Unfortunately, no method has yet been devised to assess the standard 
error for IRT or section pre-equating . Several methods have been 
proposed for estimating the standard error of NREAT linear equatings 
(Lord* 1975; Kolen, 1985). Table 9 presents for the verbal, 
quantitative , and analytical measures , the standard error of the chain of 
Tucker equatings based on the delta method developed by Lord. The median 
standard errors (assuming normality of the score distributions) are 4, 4, 
and 5 scaled score points , respectively, for the verbal , quantitative, 
and analytical measures. Thus , at least for the linear NREAT equatings, 
the smaller sample sizes appear to have. been compensated for by the 
reduction of sampling error from the use of anchor test data. That is, 
the median standard errors of the linear NREAT equatings based on samples 
of about 4,000 are about the same as the median standard errors of the 
random group , means and standard deviations equatings based on samples of 
20,000 to 30,000. 



Insert Table 9 about Here 



The RPOS equatings were based on samples approximately one-half the 
size of the NREAT samples. Could this explain the particularly poor 
performance, at least in the case of the verbal and quantitative 
measures* of the RPOS equating methods? Sinceiiostandard errorscbuld 
be estimated for the RPOSequatings^ this was assessed by dividing each 
of_the_NREAT samples in half ^ performing Tucker and Levine equatings on 
each half-sample, and estimating the bias and root mean squared error for 
each chain of half-sample equatings. Although three of the four 
half-sample equating chains had greater bias and root mean squared error 
than did their respective full-sample chains (unexpectedly, one of the 
Levine half-sample equatings had smaller bias and root mean squared 
error) , all four had considerably smaller bias and root mean squared 
error than the RPOS equating chains i 
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In summary, it appears that the performance of the NREAT arid RPOS 
equating methods relative to the standard error of the operational RG 
equatings cannot be explained by the sample sizes used in this study. 

Effect of sampling error . Six different sets of data were used for 
the equatings in this study. That is, .different examinees made up the 
samples for the NREAT and RPOS groups for each of the three GRE General 
Test measures. Thus, the results of the 18 equating chains presented in 
Tables 6_ through 8 are based on only six independent sets of data. : 
Still, of those 18 chains, only one (Tucker for the verbal measure) has a 
root mean squared error smaller than the theoretical standard error of a 
comparable chain of RG equatings based on total samples of 30,000 
examiriees. It appears highly unlikely that chance in the selection of 
samples explains these results. 

EffecL of - rea l dat a . All statistical, models, iricludirig equating 

models,. are based on assumptions that are riot strictly met by the data. 

Thus, standard errors of equating, which are based on theseunmet 

assumptions, are usually unrealistically small compared to_ corresponding 
root mean squared errors that are empirically derived. The magnitude of 
the discrepancy between thestaridarderror of equating and the 
empirically derived_root mean. squared error will depend on the magnitude 
of the discrepancy. between the assumptions and the data. The assumptioris 
of-the various equating models used in this research are given in 
*PP en ^^ x Some of these assumptioris are untes table, given the 
available data: for example, the assumption that the regression of total 
test on equating test for test X in population Q_( the population that_ 
tdbk test Y) is z the z same asthe regression of test X on the equating test 
for population P. Forbtherassumptions, such as the local independence 
assumptionof lRT 1= gobd methods of testing the assumptions did not exist 
at_the_time_this research was carried out. Previous research has 
demonstrated the reasonableness of the three-parameter logistic model for 
the GRE verbal and quantitative measures (Kingston anc Dorans, 1982a). 
Analysis of item-ability regressions (for an example of such an analysis 
see Kirigstpri: arid Dpraris^_ 1985) anda slightly modified Yen's Q: statistic 
(see Yen, 1981, 1984) indicated that the three-parameter logistic model 
is probably reasonable for the current GRE General Test. 

__ _ One assumption buried in RPOS equating is that examinee responses 
to items will be the same when the items appear in the preoperational 
sections and wheri they appear iri the operational test* .Examinees ! 
responding behavior might vary* for example* if theyknew that the 
preoperational items did not. count toward their score and therefore they 
decided riot _ to_waste_tob much time and energy on those itemsi More 
generally^ behavior might vary if there were any kind of context or 
location effect, perhaps caused by fatigue or practice in one setting, 
but not in the other (Kingston arid Doraris, 1984; Whitely and Dawis, 1976: 
Yen, 1980). 

In this research, preoperational data were always gathered in the 
seventh (last)_sectiori of the test. These "preoperational" dat* were for 
test material from a previously administered edition (that is, the old 
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fbrm_in the equating relationship) in _whiehthe__ items always appeared 
prior to the last. section* The considerable _negative_bias_f band for the 
verbal and analytical RPOS equatings indicates that for a given raw 
score, scaled scores for El when administered operationally in April 1983 
were lower than for El when administered "preoperationally" in December 
1981. For two editions of a test, the scaled score for a given raw score 
will be higher for the more difficult edition. Therefore, the verbal and 
analytical items appeared more difficult when they were administered in 
section 7 than when they were administered operationally in sections I 
and 2 (verbal) or 5 and 6 (analytical) • Note, however^ that this effect ^ 
which appears clear from the equatings, is not clear from the mean deltas 
presented in Table 1. 

Some coaching schools have advised examinees to determine which 
section or sections of a standardized test do not count toward their 
score andsave their energy by not working _tbo_ hard on those sections 
(Owen, 1985 pp* 135—136) • If a large number of examinees followed this 
advice, it might explain the results for the verbal and analytical 
measures. This potential explanation is weakened considerably, however, 
because it is unclear why this would occur for those measures but not for 
the quantitative measure. Alternatively, it might be that verbal items 
(particularly reading comprehension) and analytical items are susceptible 
to a fatigue effect. If an examinee's attention span diminishes at the 
end of a long test, this might affect, in_particuiar , itetas that refer to 
relatively long passages or that require the juxtaposition of diverse 
elements of a question. Such an effect would be more likely to influence 
responses to reading comprehension, analytical reasoning, and logical 
reasoning items than other item types. 

Previous research on the effect of item location on IRT parameter 

estimates and equating results has been. perf ormed_on the_pre-October 1981 
version of the GRE General Test (Kingston and Dorans, 1982b, 1984) That 
research was based on an RPOS data collection design very similar to the 
one used in the current study, but it differed in that the test was 
administered under formula-scoring instructions , the verbal measure was 
slightly longer and more speeded , and the analytical measure had two 
additional item types and only a few logical reasoning items. Kingston 
and _ Dorans found that location effects were item-type specific. .Analysis 
of explanations and logical diagrams items, two item types no longer used 
in the analytical measure , showed very large practice effects; that ±s t _ 
they were considerably easier when answered after another section of such 
items. Reading comprehension items were mote difficult for the examinees 
when they appeared in the preoperational section at the end of the test 
(the mean difference between b-estimotes was .14). Although for two 
different test editions the magnitude of the effect was consistent, it 
was statisticaliysignif icant at the i05 level in_oniy onei Analogy^ 
antonym, and quantitative comparison items all appeared easier in the 
*;?0S position in both editions of the test, but the differences were 
statistically significant at the .05 level only for antonyms and only in 
one of the two test editions. Because too few logical reasoning items 
were administered* no results for that item type are available. Results 
for the other item types were inconsistent. Given the change in scoring 
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directions and the inconsistency of these results, the Kingston ahd_ 
Dbrans study does hot appear to shed too much l**ght on the RPOS equating 
results. 



----- --While Kingston and Dorans did not find consistent statistically 

significant results for the item types that constitute the verbal and 
analytical measures of the current General Test , item location effects 
caused by fatigue or practice cannot be ruled but as an explanation fbr_ 
the RPOS equating results. Even if location effects were. too _ small to be 
found statistically significant, given the power of the statistical test 
that_w6uldBe_used f the effects might be consistent, and the sum of such 
affects over six equatings might be large enough to explain the bias iti 
verbal and analytical RPOS equatings. 

Comparison of individual SPE and Tucker equati ngs. Since for this 
research the preoperational and operational administrationof test items 
involved different populations^ it is nbtpbssible to assess directly the 
magnitude and. consistency of any item location effects. One way to 
assess this indirectly would be to compare each of the six SPE and Tucker 
equatings in che chain for each measure. This is reasonable for the 
verbal and analytical measures, since for them the results for the Tucker 
model were quite good. These results are presented in Table 10. Since 
for the quantitative measure the Tucker results were not satisfactory, 
such a comparison will not be presented. 



Insert Table 10 about Here 



Table 10 shows that for four out of six of the verbal equatirigs, 
SPE showed a large negative bias compared to Tucker litems appeared more 
difficult in their preoperational administration. than in their 
operational administration). For the other two equatings, the bias was 
positive but smaller in magnitude. SPE shows a negative bias in five of 
the six analytical equatings. For the E4 to E3 and E6 to E5 equatings 
the bias is particularly large, 12 and 16 scaled score points, 
respectively. The one instance of positive bias was also large, about 10 
scaled score points. 



_ _ Comparison of SFE and linear ^random- group igquating^ . For three of 
the_ equating links, E6 to E5, E3 to E2, E2 to El, there exists for 
comparison the operational RG linear equating. Such an equating is 
performed on equivalent samples from a single population and is based on 
fewer asssumptions than any of the other equating methods presented in 
this report. Table 11 presents the bias and root mean squared, error for 
the verbal* quantitative, andanalyticalSPE and Tucker equatings, using 
the linear random group_methbd as a criterion. It should be noted that 
thiscriteribn is a relative one. That is, the linear random group 
equating suffers from some amount of sampling error, it may be more 
population dependent than some other equating methods > and it assumes 
linearity of equating relationships (as does SPE). 
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insert Table H about Here 



From Table 11 It appears that for the verbal measure, bias was 
Introduced in the E3 to E2 section pre-equating and not in either of the 
other two equatings. For the quantitative measure, it seems that a 
moderate negative bias is introduced intwo of the three equatings, but a 
large positive bias is introduced in the E6 to E5 equating. In fact, for 
all other linear NREAT and RPOS quantitative equatings (data not 
presented here), the E6 to E5 equatings had large positive bias (between 
8 and 1 1 scaled score points) and root mean squared error (between 9 and 
11 points) compared to the linear RG equating. 

Data for the E6 to E5 equatings were studied closely to try to 
ascertain why they appeared to produce so much bias. The two test 
editions^ were matched well on difficulty and had essentially equal 
reliability. Test analysis data from the original administrations of the 
two editions confirmed their statistical parallelism (Wallmark, 1982, 
1984). The samples for the NREAT equatings were well matched with regard 
to anchor test score distributions. Although no causeforthepoor 
results for the NREAT equatings is apparent^ Table 3 provides evidence as 
to the possibie_culprit„for theRPOS results; The mean quantitative 
scores bnedition E6 administered in February 1983 were different for the 
two groups who took the "preoperational" half versions of E5. The means 
of 526 and 534 differ by about 2*6 standard errors of the mean: a 
difference that should occur less than one time out of one hundred by 
chance. The standard deviations for the two groups also differed: 132 
versus 129. Although these differences may well be due to chance, this 
may have affected the quality of the equatings. 

For the analytical measure, a very large amount of negative bias 
was introduced in the E6 to E5 SPE equating (about 15 scaled score 
points). Also, a moderate amount of bias was introduced in both the SPE 
and Tucker E2 to El equatings. No cause for this bias is apparent. Less 
bias was introduced in the E3 to E2 equating. 



SUMMARY AND CONCLUSIONS 



NREAT equating, both IRT based and traditional , worked well for the 
GRE verbal measure. The average estimated bias for the four equating 
methods was about 4 scaled score. points and the root mean squared error 
was about 5 points. This appears to be small, given that the verbal 
measure scaled score standard error of measurement is about 34 points 
(ETS, 1985a), and that score users are advised not to make distinctions 
between individuals based on small differences. Also, the NREAT verbal 
measure root mean squared error is about the same size as the theoretical 
value for the operational RG General Test equating method* NREAT 
double-part score equating forthe verbal measure might well be 
psychometrically somewhat superior to the current GRE equating method . 

For the quantitative and analytical measures, the NREAT equatings 
worked somewhat less well. There was essentially no bias over the chain 
of six equatings for the analytical measure , but the root mean squared 
error, depending on the equating method, varied. between 5 and 13 s-aled 
score points (average root mean squared error was 8 points). Since there 
is rib discernible bias* groups of examinees would not be disadvantaged 
because of the test edition they happened to take. The random error, 
however^ would be expected to inflate by about 20 percent the alternate 
form standard error of measurement beyond the internal consistency based 
estimate. This is somewhat larger than would be_ expected. to_ occur with 
the current operatibrial equating method. NREATdouble-part score 
equating is unlikely to be psychometrically superior to the current 
RG-based equating method for the analytical measure. 

The quantitative NREAT equatings had a positive bias of about 12 
scaled score points which accounted for almost all of the root mean 
squared error. This bias is equal to about 30 percent of the 40-point 
standard error of measurement (compared to about 21 percent for the NREAT 
verbal bias). The bias arid root meansquared error found for the 
quantitative NREAT equatings is considerably larger than those expected 
to occur with the_operational_RG-based equating method, and there is no 
reason to expect that double-part score NREAT equating would be 
sufficiently better to justify its use solely on its psychometric merits. 



RPOS equating worked poorly for the verbal and analytical sections 
because of a large negative bias. That is, tests equated from data 
gathered in the seventh (last) section of the General Test appeared more 
difficult than when they were operationally administered. There was no 
such bias, however, for the quantitative RPOS equatings. For the verbal 
measure the estimated bias was larger for the IRT equatings than for the 
SPE equatings (27 scaled score points compared to 17 scaled score 
points), but for the analytical measure the bias was larger. for_ SPE than 
for IRT (28 points compared to 17 points). These biases are fairly large 
compared to the standard error. of measurement: an average of 72 percent 
of the standard error of measurement for verbal and an average of 45 
percent for analytical. It also appears likely that these biases would 
continue to propagate and that the verbal and analytical score scales 
would drift considerably over time. 
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The quantitative RPOS equatings appeared tb^wbrfc reasonably well. 
SPE^ fbr_ the quantitative measure, had a bias of 7 scaled_score points, 
which was small compared to the NREAT biases, as was the root mean 
squared error of 10 points. The IRT RPOS equatings had no discernible 
bias, but a root mean squared error of 15 points. 

- Summary In summary , only the verbal NREAT equatings arid -none of 
the RPOS equatings appear to work as well as the operational RG equating 
works in theory. But all of the NREAT equating methods presented appear 
to work acceptably well for the analytical measure^ and the RPOS _ 
equatings appear marginally acceptable for the quantitative measure. 
Note, however, that the RG methods have not been subjected to an 
empirical check as have the NREAT and RPOS methods* It is almost certain 
that RG equating methods wil 1 not work quite as well in practice as they 
do in theory. The important question is whether they work better than 
the NREAT and RPOS equating methods. 

Now that the possible cause of bias in the RPOS equatings has been 
removed , that is, the constant use of section seven for experimental 
items, it is likely that RPOS equating methods (based on careful 
selection of experimental sections in order to balance position) will 
work better. 



Recommendations Two additional research studies are recommended* 
First, bias and root mean squared error for linear and equipercentile 
equating using the operational RG data collection design with a six-link 
equating chain should be designed and performed. The proposed study 
would provide a meaningful comparison for the results in the current 
study. Such a study is also recommended by the ETS Standards for Quality 
and Fairnesa (ETS, 1983, pi 17), which specifies that testing programs 
should, "Periodically assess the results of methods used to achieve __ 
comparability of scores and evaluate the stability of the score scale . M 

Item location effects appear to be a thornier problem than previous 
research has indicated* Additional research leading to models that 
account for such effects are needed. Such models would increase our 
knowledge of test-taking behavior and ultimately lead to fairer, more 
accurate test scores. 
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Table 1 

Description of teat Editions 



Test 

Edition 
Verbal 

E 1 

E 

E 

E 

E 

E 

E 



2 
3 
4 
5 
6 
t 



Admin. 
Date 

10/81 
12/81 
2/82 
4/82 
10/82 
2/83 
4/83 



Quantitative 



Analytical 



Tof 
Heme 

72 
76 
76 
Z6 
76 
76 
72 



Operational, 



1st Anc hor : 



Mean 
Delta 

12.0 
11.9 
11.8 
11.9 
21.8 
11.9 
12.0 



S.D. 
Delta 

2.7 
2.4 
2.5 
2.6 
2.9 
2.6 
2.7 



f of 
I tens 



38 
38 
38 
38 
38 
38 



Mean 
Delta 



11.9 
11.9 
12.0 
12.1 
11.8 
12.2 



S.D, 
Delta 



2.4 
2.4 
2.5 
2.7 
2.5 
2.3 



It ens 

38 
38 
38 
38 
38 
38 



2nd Anchor 



Mean 
Delta 

12.1 
11.8 
11.9 
12.3 
11.8 
11.9 



S.D; 
Delta 

2.6 
2.3 
2.5 
2.6 
2.6 
2.5 



_ : 1st Preop. __ 
# of- Mean S.D. 



_ l 2nd Preop. 

f of Mean S.D. 



^e»a Delta Delta I teas Delta Delta 



36 
38 
38 
38 
38 
38 



11.9 
11.8 
12.1 
11.8 
12.1 
12.2 



2.5 
2.3 
2.7 
2.8 
2.7 
2.5 



36 
38 
38 
38 
38 
38 



11.8 
12.0 
11.8 
11.9 
11.6 
12.2 



2.4 
2.2 
2.2 
2.6 
2.6 
2.7 



E 


1 


10/81 


60 


11.3 


2.7 








30 


11.1 


2.6 














E 


2 


12/81 


60 


11.3 


2; 7 


30 


10.8 


2.7 


30 


11.3 


2.6 


30 


11.1 


2.6 


30 


11.1 


2.8 


E 


3 


2/82 


60 


11.4 


2.5 


30 


11.3 


2.4 


30 


11.2 


2.7 


30 


11.6 


2.8 


30 


11.2 


2.6 


E 


4 


4/82 


59 


11.2 


2.5 


30 


11.2 


2.9 


30 


11.3 


2.7 


30 


11.4 


2.4 


30 


11.5 


2.6 


E 


5 


10/82 


60 


11. 3 


2.6 


30 


11.5 


2.7 


30 


11.2 


2.5 


30 


11.4 


2.4 


29 


11.2 


2.6 


E 


6 


2/83 


59 


11.0 


2.7 


30 


11.2 


2.5 


30 


11.4 


2.7 


30 


11.4 


2.* 


30 


11.2 


2.5 


E 


1 


4/83 


60 


11.3 


2;7 


30 


11.3 


2.6 








30 


10.8 


2.7 


29 


ii.i 


2.7 



E 


1 


10/81 


45 


13.2 


2.1 








25 


13.4 


2.2 














E 


2 


12/81 


50 


12.4 


2.2 


25 


13.3 


2.2 


24 


12.7 


2.0 


24 


12.7 


2.0 


21 


13.1 


1.6 


E 


3 


2/82 


50 


12.6 


2.0 


24 


12.6 


2.2 


25 


12.4 


2.2 


25 


12. i 


2.3 


25 


12.6 


2.1 


E 


4 


4/62 


50 


12.7 


2.3 


2D 


12.6 


2.2 


25 


12.7 


2.6 


25 


12.7 


1.9 


25 


12.8 


2.0 


i 


5 


10/82 


50 


12.5 


2.0 


25 


12.9 


2;7 


25 


12.7 


2.1 


25 


12.5 


2.3 


25 


13.0 


2.3 


E 


6 


2/83 


50 


12.9 


2.1 


25 


12.8 


2.2 


25 


12.7 


2.8 


25 


13.4 


1.9 


25 


12.3 


2.9 


E 


i 


4/83 


45 


13.2 


2.1 


25 


12.8 


2.7 








25 


13.0 


2.2 


25 


12.9 


1.6 



*?chbr - external anchor te?t 
Preop* ~ preoperational section : 

Delta - ite« difficulties on delta scale, equated within CREJieaaure _ 

Deltas for operational ite«s are based on the ltesi an^Jvsls prepared the first 
tine that the test edition was administered; thus the statistics for the two 
adsilnistrations of El are the aaae. Deltaa for the anchor teat and preoperational 
sections are baaed on their adsilniat ration as part of this research. 
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Table 2 

Description of Samples Used for Verbal Equacings 









N 




N 




N 




N 


Test 


Admin. 


1st 


X 


2nd 


X 


_lst 


X 


_2rid 


X 


Edition 


Date 


Anchor 


s 


Anchor 


s 


Prebp 


• s 


Prebp 


• s 












4*408 










El 


10/81 






Al 


500 
































- 





- - - - - 





126 
















4,096 




4, 180 




2,062 




2,076 


E2 


12/81 


Al 


473 


A2 


477 


Ela 


476 




Eib 


475 








123 




122 




121 




122 








3*746 




3*602 




1,808 




1 *760 


E3 


2/82 


A2 


484 


A3 


484 


E2a 


483 


E2b 


480 








119 




118 




120 




120 








3,647 




3,604 




1,789 




1 ,747 


E4 


4/82 


A3 


463 


A4 


462 


E3a 


464 


E3b 


465 








124 





- 125 




125 


- — 


125 








4,331 


- - 


4,230 


- - 


1,713 




1,654 


E5 


10/82 


A4 


518 


A5 


516 


E4a 


523 


E4b 


522 








127 




126 




126 




125 








0 DOC 




J * 0/ 1 




1 , 808 




1 , 9U4 


E6 


2/83 


A5 


478 


A6 


478 


E5a 


478 


E5b 


482 








_ 121 




-110- 




124 




12Z 








4,209 








2,083 




2,073 


El 


4/83 


A6 


474 






E6a 


475 


E6b 


477 








125 








127 




126 



Anchor - external anchor test 
Preop. - preoperational section 



_____ ... .._ . . Table 3 

Description of Samples Used for Quantitative Equatings 









N 




N 




N 




N 


Test 


Admin* 


1st 


X 


2nd 


X 


1st 


X 


2nd 


X 


Edition 


Date 


Anchor 


s 


Anchor 


s 


Preop 


• s 


Preop 


• s 












4,329 










El 


10/81 






Al 


531 
















































4,147 




4, 144 




2,068 




2,028 


E2 


12/81 


Al 


526 
133 


A2 


524 
135 


Ela 


526 
132 


Elb 


525 
133 








3,591 




3,656 




1 ,756 




1,718 


E3 


2/82 


A2 


525 


A3 


524 


E2a 


521 


E2b 


519 








131 




129 




- 132 












3,583 




3,646 




1,809 




1,769 


E4 


4/82 


A3 


502 
136 


A4 


502 
138 


E3a 


504 
137 


E3b 


498 
135 








4j338 




4-335 




1,719 




1 ^678 


E5 


10/82 


A4 


542 
134 


A5 


544 
134 


E4a 


548 
133 


E4b 


543 
136 








3,754 




3,786 




1,863 




1,816 


ES 


2/83 


A5 


529 
133 


A6 


529 
132 


E5a 


526 
132 


E5b 


534 
129 








4,095 








1,936 




2,075 


El 


4/83 


A6 


507 
136 






E6a 


510 
136 


E6b 


505 
135 



Anchor - external anchor test 
Preop. - preoperational section 



- 26 - 32 



------- Table 4 .... 

Description of Samples Used for Analytical Equatings 

~~ N N N ' N~ 



x 
s 



Test Admin. 1st x 2nd x 1st x 2nd 
Edition Date Anchor s Anchor s Preop. s Precp. 

4,357 

El 10/81 — Al 523 

130 _-- 

4,179 6,009 6,009 -■ 1,943 

E2 12/81 Al 511 A2 514 Ela 514 Elb 517 

- 125 ■ 124 124 126 

3,594 3,523 1,839 1,798 

E3 2/82 A2 503 A3 504 E2a 501 E2b 501 

126 126 125 120 

3,652 3,596 1,783 1,745 

E4 4/82 A3 489 A4 489 E3a 489 E3b 488 

128 125 125 125 

4,285 4,339 2,561 2,459 

E5 10/82 A4 525 A5 521 E4a 524 24b 525 

130 129 130 133 

3,698 3,829 1,813 1,797 

E6 2/83 A5 510 A6 513 E5a 513 E5b 51 i 

124 125 122 127 

3,945 2,011 1,940 

El 4/83 A6 502 — E6a 504 E6b 503 

. 128 128 133 



^Anchor - external anchor test 
Preop. - preoperational section 



- 27 
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Table 5 

Equating Methods Used in This Study 



Data __ 
Collection Transformation 

Design Linear IRT 



RG Mean arid S.D. 



Tucker IRT True Score 

NREAf Tucker True 2 

Levine Equally Reliable 
Levine Unequally Reliable 

RPOS SPE (EM Algorithm) IRT True Score 



34 
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___Table!_S- 

Bias and Root Mean Squared Error in the Raw Score Metric 
for Various Equating Models 



Equating Method 


Verbal 


Quari 


Lit atlVc 


^Analytical 


Bias 


RMSE 




Cxi 1«J Lj 




Ivl iO Hi 


Section Pre-equating 


-1.65 


2.31 


.50 


.74 


-1.59 


1.60 


IRT RPOS 


-2.56 


2.69 


.02 


i.i2 


-;95 


i;37 


IRT NREAT 


-.52 


.63 


1.03 


1.10 


.04 


.76 


Tucker 


-.06 


.11 


1.11 


1.11 


.30 


.52 


Tucker True 2 


-.49 


.57 


.67 


.70 


.03 


.33 


2 

Levine, as appropriate 


-.56 


.61 


.84 


.85 


-;d3 


.31 



l Itk raw score units, see text for definition 

2 1 -_ - _ _- -- --_ 

Chain of Levine equatings, using parameters based on equally reliable 

model or unequally reliable model, based on whether or nst the old and 

new editions of the test are the same length 
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: _ . .- - ...^ _ --- -- Table 7 

Bias and Root Mean Squared Error in the Scaled Score Metric 
for Various Equating Models 



Equating Method 


Verbal 


Quant i 


tative 


Analytical 


bias 


DMCt? 


Bias 


RMSE 


Bias 


RMSE 


Section. Pre-equating 


-17 


24 


7 


10 


-28 


28 


IRT RPOS 


-27 


28 


0 


15 


-17 


24 


IRT NREAT 


-5 


7 


14 


15 


l 


13 


Tucker 


-1 


1 


15 


15 


5 


9 


Tucker True 2 


-5 


6 


9 


la 


1 


6 


Leviiie* as Appropriate 


-6 


6 


11 


12 


-1 


5 



See text for definitions of bias and root mean squared error. 

2 Chain of Levine equatings, using parameters based on equally reliable 
model. or unequally reliable model, based on whether or not the old and 
new editions of the test are the same length 



- 30 - 
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Table 8 










o u ana a raizeQ 


Bias and 


Root Mean 


Squared 


Error * 


(xlOO) 






for Various Equating Models 










: - Verbal : 


Qoantf tatitve 


-lAnalyticai- 


Equating Method 


Bias 




Bias 


RMSE 


Bias 


RMSE 


~i ~ i - '. . 
Section Pre~equa ting 


-14 


20 


5 


8 


-22 


22 


1K1 KFUt> 


-22 


23 


0 


11 


-13 


18 


TRT MRFAT 


-4 


5 


10 


11 


0 


10 


Tucker 


-0 


1 


11 


11 


4 


7 


Tucker True 2 - 


-4 


5 


7 


7 


0 


4 


Levine, as Appropriate 


-5 


5 


8 


9 


0 


4 



See text for definitions of bias and_root_mean squared error , both of 
which are given in hundredths of a standard deviation to avoid decimals. 

Chain of Levine equatings, using parameters based on equally reliable 
model or unequally reliable models based on whether or not the old and 
new editions of the test are the same length. 
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Table 9 

Standard Error of Equating (in the Scaled Score Metric) 
of the Chain of Tucker Eqtiatings 
For Selected Raw Scores 





Verbal 




Quantitative - — 




Analytical 


Score 


Standard 


Score 


Standard 
Error 


Score 


Standard 
Error 


Raw 


Scaled 


Error 


Raw 


Scaled 


Raw 


Scaled 


72 


824 


9.3 


60 


825 


8.7 


45 


915 


13.7 


68 


810 


8.3 


54 


743 


6.8 


40 


828 


HA 


60 


702 


6.3 


48 


662 


5.2 


35 


740 


8.6 


52 


600 


4.6 


42 


580 


4.0 


30 


653 


6.4 


44 


510 


3.5 


36 


499 


3.9 


25 


565 


4.9 


36 


427 


3.7 


30 


417 


4.8 


20 


478 


4.8 


28 


353 


4.9 


24 


336 


6.3 


15 


391 


6.2 


20 


272 


6.7 


18 


254 


8.1 


10 


303 


8.4 


12 


185 


8.7 


12 


172 


10.0 


5 


216 


10.9 



-n- 3 8 



. -_- : Table 10 : _.. 

Weighted Mean Scaled Score Difference (Bias) 
Between SPE and Tucker Equatirigs 
for the Verbal arid Arialytical Measures 



Equating 


Weighted Mean Scaled ! 


Score Bifference 


Verbal 


An^'/tieal 


E2 — > El 


+1 


-3 


E3 — > E2 


-6 


-7 


E4 — > E3 


-7 


-12 


E5 — > E4 


+3 


+10 


E6 — > E5 


-4 


-16 


El — > E5 


-7 


-3 



ERIC 
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^Table_ll^_ 

Scaled Score Bias and RMSE of SPE and Tucker_Equatihgs 
Using Linear Random Groups Equating as a Criterion 



Verb al^ — Quantitative Analytical 









SPE Tucker 


SPE 


Tucker 


SPE 


Tucker 


Bias 


















E2 


— > 


El 


0 


-2 


-3 


4 


5 


8 


E3 


— > 


E2 


-6 




-4 


-3 


-1 


5 


E6 


— > 


E5 


1 


4 


9 


11 


-15 


2 


RMSE 


















E2 


— > 


El 


3 


2 


3 


4 


7 


9 


E3 


— > 


E2 


6 


2 


4 


6 


2 


6 


E6 


— > 


E5 


1 


5 


9 


11 


15 


2 



40 



Admin. 
Dace 



Figure 1 
NREAT Data Collection Design 



Operational Test Editions and External Anchor Tests 



10/81 
12/81 
12/81 
2/82 
2/82 
4/82 
4/82 
10/82 
10/82 
2/83 
2/83 
4/83 



El 



Al 



Al 



E2 



E2 



A2 



A2 



E3 



E3 



A3 



A3 



E4 



E4 



A4 



A4 



E5 



E5 



A5 



A5 



E6 



E6 



A6 



A6 



El 
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Admin. 
Date 



12/81 
12/81 
2/82 
2782 
4/82 
4/82 
10/82 
10/82 
2/83 
2/83 
4/83 
4/83 



Figure 2 
RPOS Data Collection Design 

Operational test Editions and Preoperational Sections 



El 


E6a 


El 


E6b 



E6 



E6 



E5 



E5 



E5a 



E5b 





E2 


Ela 


E2 


Elb 


E3 


E2a 




E3 


E2b 



E4 



E4 



E4a 



E4b 



E3a 



E3b 



ERIC 
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Figure 3 



Equating the 6RE- Verbal Measure to Itself 

- - through a Six— Link Chain: 
Conversion tines for Six Equating Methods 



72 



60- 



48- 



Equiydlent 
Raw Score 35 

on Et 
in 10/82 

24-I 



12- 




12 24 36 48 60 

Raw Score on El in 4/83 



LEGEND: DATA 



Cr j te r ion 

IRT NREAT 

3RT RPQS 

Leyine 



SPE : 

--• Iucke r 

Tucker 2 True 



72 
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4.9 



Figure 4 



Equating the GRE Quantitative Measure to Itself 
through a Six-Link Chain: 
Conversion Lines for Six Equating Methods 



60-1 



50- 



40- 



Equivalent 
Row Score 30- 

on El 
in 10/82 

20-j 



10- 




Raw Score on E1 in 4/83 



LEGEND: DATA 



Cr i ter Jen 
IRT NREAT 
IRT RPOS 
Lev i ne 
SPE 

Tucke r 

Tucker 2 True 



9 
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Equating the GRE Analytical: Measure to Itself 
Through a Six— Link Chain: 
Conversion Lines for Six Equating Methods 




■ ' ■ 1 ' ' s 5 ■ ■ ' ■ i ■ — ■ — ■ — ■ — ■ — i — ■ — ■ — « — ■ — ■ — I — ■ — » — ■ — ■ — ■ — r 
0 9 18 27 36 45 



Raw Score on El in 4/83 

LEGEND: DATA Criterion 

— - IRT NREAT 

IRT RPOS 

L e v i ne 

SPE 

Tuejcer 

Tucker 2 True 



" 39 - 45 



8-1 



4- 



Figurc 6 

ng the GRE Verbal Measure to Itself 
Through a Six— tink bhain: 
Differences for Six Equating Methods 



Raw Score 
on El in 4/83 
Minus o 
Equivalent 
Raw Score 
on El in 10/82 




-4- 



-64 



'i 1 ' 1 1 1 i i I i 

12 24 36 48 60 

Raw Score oh El in 4/83 



LEGEND: DATA 



IRT NREAT 

IRT RPOS 

Le v i he 

SPE< . 

-- Tucke r 
— Tucker 2 True 



72 



- 40 46 
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Figure 7 

Equating the SRE Quantitative Measure to itself 
z _Througk a Six— Link Chain: 
Differences for Six Equating Methods 



4-3 



2- 



- _ _ __ ITT 



Raw Score 
on El in 4/83 
Minus q_ 

Equivalent 

Raw Score 
on E1 in 10/82 



-2- 



;4= 



16 20 30 40 50 60 

Raw Score oh El in 4/83 

LEGEND: DATA IRT "RE AT 

IRT RPOS 

Levin* 

SPE 

-- :: - Tucke r 
Tucker 2 True 
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Figure B 



Equating the GRE Analytical Measure to Itself 
Through a Six— Link Chain: 
Differenced for Six Equating Methods 

4 



2 



Raw Score 
m El in 4/83 
Minus o 

Equivalent 

Raw Score 
n El in 10/82 



-2- 



-4 

i — ■ — ' — ■ — ■ — ' — i — 1 — ■ — ' — ■ — r— r ■ — ■ — ■ — ■ — ■ — i — ■ — ■ — ■ — ' — ■ — I — ■ — ■ — ' — ■ — —r 
0 9 18 27 36 45 

Raw Score oh El In 4/S3 

IKT NREAT 
IRT RPOS 
Lev ihe 
SPE 

Tucke r 

Tucker 2 True 
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LEGEND: DATA 



Appendix B 



Linear Equating Models 
Y - AX + B 



Notation and Computational Formulas 



Reproduced from Marco, Petersen, and Stewart (1982) 

B-l 

51 
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Notation 



New Test Form X 

Old Test Form T 

Either New or Old test Form P 

Anchor test V 

Observed Score x, y t v, p 

True Score x*, y* , v* t p* 

Error Score x'\ y'\ v" t p" 

Group taking Test X and Test V a 

Group taking Test t and test V b 

Either Group taking X and V or Group 

taking Y and V g 

Combined Group c or (a+b) 

Mean M 

Standard Deviation S 

Covariance C 

Part Score x- , y. , v., p. 



B-2 
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TPCKE* 1 
Estaroal or Xocarnal Anchor 



TUCKE* 2 TREE 
Excarnal Anchor 



faraactart for Kalatlhj lav Scbros 



NrtMtin for lalatlnt lav Scor«» 



2 



Additional Sf tlitlo 
Rogrcaalon of p on 



Add 1 1 1 ona jrS tartaric* 
lagraaaloo of p' on 



Slooa • W - C /S 

pV g pVg T g 



Intercept • H-- - © M 

PI P*l *g 



Slop* - V . .- • C /(S 2 - S 2 M ) 
P » I P*l *g *"g 



Intercept - H- - - V . -,-H 
PI P * I 



Eatioatat for Croup c on p: 



Eatlaetee for Croup c on p' 



Heen «H . _-H ) 

PC pg pv| *c V| 

2 "2 J "J "5 

Variance - - S ♦ tT (SV - ) 
pc pg pv g vc vg 



Main ■ M • • H ♦ . CM- - M- ) 
P c PI P * I *c Vg 

Varlanca - S 2 - ($ 2 - S 2 „ ) ♦ v 2 . . (S 2 - S 2 ) 
P e pg p"g p v g vc vg 



IE VINE OV EQUALLY RELIABLE 
Extaraol or lotarnal An en or 

■ rs Tor^aiettne— lav- Score* 



t(S J - - S 2 „ >/(S 2 _ 



2 v1 l/2 



• M L - AK + <M • H JUS* - S V>/(S\ - S*,» )) 
yb xe va vb yb y"b vb v'V 



Additional Stetlftic t 



legreieion of p' on ▼': 



51dpi - W . - t(S 2 - S 2 M >/(S 2 - S 2 „ )J l/2 
P * I PI PS ▼g v M g 



Utarcape - * p% - * 



Eatlaetee for Croup c on p': 



MaaD ■ H . • H ♦ V . . (M - H — ) 
P c pg p V g vc vg 

Varlanca - S 2 . - V 2 . . (S 2 - S*„ ) 
p c p v g v C v M g 
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LEVXNE EQUALLY RELIABLE 
External or Intarnal Anchor 

Firintin for lal at ln* Haw Scoraa 

* • 4 * (s U - s o'«« - s 'b"< - s 'v il/2 
[s L * (s L * s «"« Hs « * s v. ,/(s v. • s l", )] ~ l,i 

1 • H yb * <*vc " H vb >US lb " ^"b^ 5 '* " S ,\ ,?! ' 2 

Additional Statistic* 
Ragraulon of p' on v': 

Ine.rc.p. - « pg - y,.-",, 

T.*:inet«« for Croup c on pt 

Haan • M • Kh 1 ♦ V . . - (M - M ) 
PC PI P v | vc vg 

Varlanca • S 2 ■ ♦ W 2 . . (S 2 - S 2 ) 
pc p| p v g vc vg 
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TUCKER 2 OBSERVED 
Estaraal Anchor 



TUCKER 3 OBSERVE* 
Estaroal Anchor 



firwttri for Ralatlne Rg* $ c0 Tf » 



Nriwtiri for K«l«tlog R«v Scort« 



ft*dlt*on*l— St» 1 1 1 1 1 c ■ 
R«grt»*loo of p on ▼': 



Addition al t«tl«tlc» 
Ragratalon of ▼ on p'; 



Slopi • W -,- • C /(S 2 - S 2 „ ) 



Slop* - W . • c --/($-- - S 2 „ ) 
*P I pvg pg p M g 



Intercept » ► - - V , M 

PI P v I *g 



Xnttrctpc • M^- - W_ . h 



*? g PS 



Eatlaettt fcr Croup c on p: 



EetlAataa for Croup c oo p: 



Mean * H * M ♦ V . (K - M ) 
Pt pg p*'g *c vg 

Variance • S 2 • S 2 ♦ W 2 . (S 2 - S 2 ) 
pc pg pv | v C vg 



Maab - M • M ♦ Cm - h )/U- 
Verlanca - S 2 • S 2 ♦ (S 2 - S 2 )/U 2 . 

PC Pg *C Vg" Vp'g 



TUCKER 3 TRUE 
External Anchor 



TUCKER MODIFIES LEVTNE 
External Anchor 



Ferametera for Relating Rev Scoree 

,2 .2 



Faramatar* for Ralatlne. Raw Scor aa 



A * I k /H _ 



* • H^w * B-^CM - H k ) - AM 

VI Vfi Zt 

vhara B - (1-K B >C/2S 2 ♦ Ul-K ) 2 C 2 ♦ 4S 2 S 2 K J 1/2 /2S 2 
Pf PI P*g *g pg pvg vg pg pg" " a v| 

- Vtlghc 



Atfdtnorjlr StatHtjc t 
Regreeeion of v* Q a p"; 



K . (S 2 - - S ? tt )S 2 1 , L /S 2 i, (S 2 - S 2 „ ) 
Pf pg p g v"g pV vg *V 

* Ralaclvn Effactlva Taac Length 



* p • pvg /l pi Vi 



Intarcapt - M - W , , «- 
' P I Pg 



Additional _Statlartc* 

Eattmatta for Croup c on p*: 



Estimate* for Croup c on p* 



Haan • H . • n - ♦ 8 (M - « - ) 
p c pg pg rc vg' 



Mean * m . - M ♦ ft* - - M- )/W ~ # -.- 

P C pg V C Vg' V*p'g 

Vtrlanca • S 2 . - (S 2 - S 2 ,. ) ♦ ($ 2 - S 2 )/V,' 2 . : 
P c Pg P I vc a vg" v'p'g 
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«iriinc« - S 2 . • e 2 (S 2 - s 2 „ ) 
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LORD XT 
External Anchor 



?irwtin for Relating Rgv Scoraa 



«tiar« H- - (S 2 - S 2 « )/C • Weight 
PI PI PI P*I 



Additional Sci tlatlce 



Adjuated Maao of p for Croup g - M - H M 
* PI PI 



Error Varlaoca of v for Croup g • S 2 ,, * - C /B 

▼ i n p y t pi 



DID V 
Ei carnal Anchor 



Peregetere for ItaUtlo 



A - B 

yb sa 



i . ft ♦ b k (n - n . ) - am 

yb yb *e vb x 



Additional Stetlexlci 



Adjueted Meeo Of p for Croup g - M - B _H 

Y * PI PI V I 

2 2 

Error Verlence of p for Croup t - S „ - S - C P 

pi pi p f g pg 



LORD HAJC-TTC LIKELIHOOD 
Exteroel Anchor 



LORD CONGENERIC SUBTESTS 
External Anchor 



Rereoecera for Re la t ion R«v Sco r e * 



geran etere for Relet Inn Raw Scoraa 



A - B W /B 
yb & 



S " H yb * B yb ( %n " M vb> - *»x 



yb yb wm Yb sa 



where B - (S 2 - Q S 2 ) /2C ♦ t(S 2 - O S 2 ) 2 ♦ 40 C 2 l l/2 /2C 
PI PI H I v g pvg pt % pvi 1 



pvg 



hare B - (C - S )/C (I - C /C 1 
PI PP!l Pjl ' PjU P^I pvi } 



Walght 



- Walght 



Q - S Z „ /S 2 „ 
I PI * I 



Additional Statletjca 



■ Ratio of Error Veriahcee 



Adjuatcd Maen of p for Croup g - n - B M 

pi pi *i 



Adjuated Maan of p for Croup g • M-- - B M 

PI PI vg 

Error Varlaoca of p for Croup g • S 2 M : • S 2= - c B 

PI PI P V B PI 

Error Varlaoca of v for Croup g - S 2 • S 2 - C = /B 

* I U pvi PI 
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fotrfiorr ac 

Ex tarn* I or Internal Anchor 



fara»eter i for talatlnt taw fenw 



A - B— /B , 
xa yb 



nTTBsrr i 

External or Internal Anchor 
for »«latlne K«w SCorea 



A - ■-/! - 
yb 



- viijht 



D It positive aquarct root of the eolutlot. for 



D of 



vfccre B - C - IS- - Weight 
PI P V B PB 



Additional Sci tlatlcs 



Adjusted Mean of ▼ for Croup a » H - B H- 

T B Pt Pf 



N ( la else of Croup i 



Additional Startnrtra 



Adjusted Haan of v for Croup ft - DM - B M 

V B PS PB 



poTTBorr t> 

Ex carnal or Internal Anchor 
gwTarsrtarra for Relet lea Raw Scorie 



A - H- AB r 
xa yb 



yo ▼ b yb ma 



Addltlcnil Stotlttlct 



Adjusted ha*t, of » for Croup I - H - I H 

»• PI PI 



9 
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Appendix C 

Notes on Other Equatings 




I- methods 



An additional ten linear anchor test equating methods were used_iri 
early stages of this study (Tucker 2 Observed Score, Tucker 3 Observed ... 
Score, Tucker 3 True Score, Tucker Modified Levine, Lord XY, Lord V, Lord 
Maximum Likelihood, Lord Congeneric Subtests, Pothoff AC, and Pothoff D). 
None of these methods appeared to work as well as those discussed in the 
body of this report. Readers interested in the results of a 
comprehensive study of these equating methods are referred to the work of 
Petersen, Marco, and Stewart (1982), 



Reliabilit " arid Levine equating 

Using KR-20 based reliability estimates , the. Levine unequally- 
reliable-tests method did as well as or slightly better than choo< ing 
between the two Levine methods based on equality or inequality of test 
length separately for each of the six equating links. 

Considerations regarding the equating criterion 

It has been argued that equating a test to itself through a chain _ 
of other test editions might favor linear equating methods over nonlinear 
methods^ because the criterion equating, an identity function, is linear. 
To avoid this, one could equate El to E2 to E3 to E4 and also El to E6 to 
E5 to E4 and compare the two composite El to E4 equating functions. 
Agreement between the functions might be a better criterion, on the brie 
hand, because the true equating relationship between El arid; E4 is net 
neccesarily linear. But , on the other hand , the true equating 
relationship for this criterion is unknown. The two equating functions 
should be the same, but even if they are the samethey could be 
consistently wrong. For example, ignoring all data and choosing the 
equating function X = Y would give perfectly consistent results for all 
equating chains. 

Bias and root mean squared error were calculated for five linear 
equating models (Tucker , _ Tucker 2 True , Levine Equally Reliable , Levine 
UnequallyReliable, and SPE) using the two-chain criterion^ If linear 
methods are favored by the criterion used when a test is equated to 
itself, then bias and root mean squared error for the linear models 
should have been larger for the two-chain criterion compared to the 
criterion. This did not occur. For all models except SPE, bias and root 
mean squared error were almost the same for the two criteria. For SPE, 
the two-chain criterion actually had somewhat lower bias and root mean 
squared. error than. the_ one-chain criterion. The rank ordering of the 
equating methods in_ terms of bias and root mean squared error was the 
same for each criterion. 



Setting the 1RT T heta Metric 

Most IRT equating methods , including those used in this study, 
require that item parameter estimates for the old and new editions of the 
test be on the same (albeit arbitrary) theta metric. This was 
accomplished in this study in two different ways: concurrent parameter 
estimation and least squares transformation (Stocking & Lord, 1983) . In 
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concurrent -estimation, for the NREAT data collection design,, the data for 
E l>_E2j and the anchor test commbh_tQ both editions , . Ai ; were analyzed in 
a single LOGIST ruh^ with_the E2items coded as not reached for the group 
who took Ei^arid-theEl items coded as not reached for the group who took 
E2^_ The parameter estimates were then used to equate E2 to El : . 
Similarly, E2 and E3 data were parameterized together with anchor A2 and 
then E3 was equated to E2. This was done for each of the six equating 
links and then the composite equating function, El to El; was calculated. 

For the least squares transformation method, the form of El that 
contained the Al anchor was calibrated separately as was the form of E2 
which. contained the Al anchor, then, the Stocking-Lord least squares 
transformation was used to determine the linear relationship between the 
two theta metrics. This transformation was applied to the E2 item 
parameter estimates. Similarly, the form of E2 that contained. the. A2 
anchor and the form of E3_ that. contained the A2 anchor were separately 
calibrated and the parameter estimates for E3 were transformed through 
the composite E3-E2-E1 least squares transformation to the El theta 
metric*. This process was repeated with the other test editions until all 
parameter estimates were on the El metric. Then, using the two sets of 
El estimates, both now on the original El scale, El was equated to 
itself. Similarly, both concurrent estimation and the Stocking-Lord 
procedure were used with the RPOS data. 

The two NREAT IRT equatings of El to itself based on the two 
methods of setting the theta metric produced almost identical bias and 
root mean squared error for all three GRE measures. For the RPOS IRT 
equatings, the results were almost the same for the two methods for both 
the verbal and analytical measures. Although the root mean squared error 
was essentially the same for the two methods for the quantitative 
measure, bias was larger for the least squares transformation method. 
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